An amount of information available by way of the World Wide Web has grown exponentially, such that billions of items are available by way of the World Wide Web. This explosive growth of information available on the Web has not only created a crucial challenge for search engine companies in connection with handling large scale data, but has also increased the difficulty for a user to manage his or her information needs. For instance, it may be difficult for a user to compose a succinct and precise query to represent his or her information needs.
Instead of pushing the burden of generating succinct, precise search queries to the user, search engines have been configured to provide increasingly relevant search results to user queries. More particularly, a search engine can be configured to retrieve documents relevant to a user query by comparing attributes of documents together with other features such as anchor text, and can return documents that best match the query. Conventional search engines can also consider previous user searches, user location, and current events, amongst other information in connection with providing the most relevant search results to a query issued by a user. The user is typically shown a ranked list of universal resource locators (URLs) in response to providing a query to the search engine.
Properly ranking search results is an important task, as a typical user is not willing to sift through several pages of search results, but instead only reviews a most prominently presented relatively small number of search results on a search results page, before entering a different query or abandoning the search entirely. Thus, often a searcher will assume that a small subset of search items shown on a first search results page is most relevant to the user and/or the query, when in actuality the user may have found a search item on a subsequent page that is most relevant to the user and query. Ranking of search results is performed by ranking algorithms (rankers). Information retrieval metric methods are used to determine the quality of a ranking generated by a ranker, as well as a cost of a ranking generated by a ranker (e.g., a higher measure of quality corresponds to a lower measure of cost).